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ABSTRACT 

Previous studies were conducted with 14- to 
15-year-old students on the nature of the processes they used to 
solve tasks given the incorrect/buggy /mal-rules . The computer-based 
Leeds Modelling System (LMS) was implemented and a database of 
examples, correct and incorrect, or mal-rules had been established 
which was sufficient to diagnose the majority of difficulties 
encountered by 15-year-old students. When the same database was used 
with 24 14-year-old students, the outcome was very different. A high 
percentage of the student errors were not diagnosed by LMS. These 
protocols were analyzed in some detail, followed by individual 
interviews to determine the nature of the students' difficulties. 
Results obtained from the protocols and interviews indicate that 
there are two very different types of mal-rules at large with algebra 
students, namely, manipulative (defining a manipulative mal-rule as a 
variant on a correct rule which has one substage either omitted or 
replaced by an inappropriate/incorrect operation) and parsing 
(incorrect representation of a task). Although bug-migration could be 
explained by repair theory, it appears that this second category of 
algebra errors (parsing) can best be explained by a mechanism called 
mis-generalization. (JN) 
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1. Introduction 

Intelligent Tutoring Systems (ITSs) force their implementors to be explicit about domain knowledge, 
tutoring rules, likely student misunderstandings for a particular domain, etc. Although this 
explicitness is demanding it does have the advantage that if the system behaves differently than 
expected, the implementor can determine the reasons for this, modify the suspected rule/knowledge 
and rerun the system. 1 Further once one has identified misunderstandings which one believes arise 
pretty consistently in a subject domain, by this or more conventional techniques, then a series of 
additional investigations are.possible. These include: 

1. hypothesizing the nature of the processes used by students to solve tasks given the 
incorrect/buggy/m '-rules. 

2. building a remedial subsystem which exploits the inferred student model (this will involve 
further analysis of teacher-student remedial dialogues). 

3. undertaking studies aimed at improving the initial instruction in the domain so as to avoid 
(some of) the observed difficulties. 

In this article I discuss the first of these points in the context of extensive studies undertaken with 
14- to l5-yearold algebra students. The Leeds Modelling System, LMS, was implemented and a 
database of examples, correct- and incorrect, or mal- rules had been established which was sufficient 
to diagnose the majority of difficulties encountered by 15-year-old students, Sleeman [1982]. The 
same database was then used with 24 14-year-old students and the outcome was very different. A 
high percentage of the student errors were not diagnosed by LMS. The investigator analysed these 
protocols in some detail and then carried out individual interviews to determine the nature of the 
students' difficulties, Sleeman [1983a]. The pertinent observations from this latter experiment are; 

1. Students appear to regress under cognitive load. That is they are often able to use a 
particular rule correctly in the context of simple tasks, but make errors with this same rule 
when the tasks are more complex. 2 See Sleeman [1983a] for examples. 



The approach used within the Expert Systems paradigm. 

This analysis assumes that domain rules are independent and one rule does not subsume another. 



2. There appears to be a number of clearly identifiable types of error, (section 2). 

3. Students use a number of alternative "methods" to solve tasks of the same type, (section 
3). 



2. Observed types of student errors 

From the protocols and the interviews I concluded that in this domain errors could be classified as: 
manipulative, parsing, execution/clerical and random. The first two topics will be dealt with in some 
detail in the rest of this section; see Sleeman [1 983b] for details of the others. 



2.1 Manipulative Errors 

I define a manipulative mat-rule to be a variant on a correct rule which has one substage either 
omitted or replaced by an inappropriate or incorrect operation, c.f., Young & O'Shea [1981]. For 
example, MNTORHS 3 is a mal-rule which captures the movement of a number to the other side of the 
equation, where the student omits to change the sign of the number. MXTOLHS is the mal-rule which 
corresponds to the analogous X-to-lhs rule. (Most of the errors noted with 15-year-old students were 
of this form.) Note that this schema would ALSO generate many mal-rules, which we have NOT yet 
observed; in the next paragraph we give an explanation why some of the possible mal-rules are not 
observed. 

a) Analysis of some manipulative mal-rules: A schema for generating manipulative mal-rules 

.In a recent experiment we noted three (additional) mal-rules which can be explained by this 

mechanism. Two of them will be analysed in some detail: 

1 . A variant on SOLVE . The variant on SOLVE transformed: 

4*X « 6 to X*6 

whereas SOLVE would change the.same expression to X ■ 6/4. It is suggested that the 
student realizes he has a task in which the SOLVE rule should be activated and forgets to 
apply one of the operations, namely dividing by M. SOLVE has three principal actions: 
noting down N, the divide symbol and M, and so this mal-rule could be said to be omitting 
some of the principal steps. Furthermore, it appears that students have an idea about the 
acceptable FORM of answers and so given the above task we have nor seen X = 6/ or 
X = /4. 

2. A variant on SIMPLIFY . Examples of the two mai-rules noted here, which have occurred 
reasonably frequently are: 

X * 6/4 => X = 3/4 



MNTORHS is short for mal number-to-rhs rule. 



~ X » 6/4 «> X - 6/2 

(The SIMPLIFY rule transforms the same expression to X - 3/2), / 

Again we argue that the above observations can be explained if we assume that this rule 
has several principal steps including, calculate the common factor, divide "top" by 
common factor, divide bottom by common factor, write down the components, and that 
each of these mal-rules corresponds to one step being omitted. 

bl "Grain size" and manipulative mal-rules . 

There is a sense in which detailed analyses of manipulative mal-rules allows one to infer the substep 
processed by students, and tf\is in turn allows one to predk i the set of mal-rules that will be 
encountered in a domain. (Bearing in mind the idea of acceptable form outlined above). Further, one 
might argue that the representation of the tasks should be at this "lower" level; the justification for the 
representation chosen, is.that this appears to be more consistent with the collected verbal and written 
protocols for students solving these tasks. The schema discussed above for generating manipulative 
mal-rules by omitting, or modifying, one substep is thus consistent with Young and O'Shea's 
modelling of subtraction. \ 

• V 

2.2 Incorrect Representation of the Task or Parse Errors 

I assert that many of the students whom we interviewed carried out steps of the computations in 
ways which would not fall within the definition given earlier for manipulative mal-rules. Below, I give 
typical protocols for two students working the task 6*X = 3*X +12: 

I: 6»X = 3*X + 12 II: 6*X = 3«X + 12 

9«X = 12 X + X = 12 + 3 - 6 

X = 12/9 2*X = 9 

X = 4/3 X = 9/2 

When I pressed the "first" student for an explanation of how the original equation was transformed 
into the second, i.e., 9"X = 12, the student talked about moving the 3*X term across to the left hand 
side. Thus the interviewe r concluded that this was an instance of a student using a variant of the 
correct rule, namely a manipulative mal-rule. When the "second" student was pressed he simply 
asserted that the change from the original equation to the second line "was all done in one step". 
Hence the interviewer concluded it was a very different type of mal-rule involved and no! a simple 
variant on the correct rule. Thus the interviews provided essential additional information as, of 
course, the second student's protocol could be explained by the use of MXTOLHS and the mal-rule; 

M*X = > M + X 

4 which some people might wish to argue constitutes a manipulative mal-rule (replacing the • operator 
by the + operator). Even if we did not have the additional experimental evidence, this investigator 
would maintain that such a transformation belays a profound misunderstanding of algebraic notation 
and so should be considered as a parsing mal-rule. See Sleeman [1983b] for additional discussion of 
this issue. 



Where M stands for an integer, and where in the abo- sample 6*X = > 6 + X and 3*X = > 3 + X. 
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3. Bug Migration or Using Alternative Methods 

Repair theory gives a neat explanation for the observed phenomena of bug migration in the domain 
of multi-column arithmetic, Brown & VanLehn [1980], namely that the student will use a related family 
of mal rules, and possibly the correct rule, during a single session with one particular taSK set. 

There seems to be an alternative explanation which should also be considered. Although a task- 
set may have been designed to highlight one particular feature, the student may spot completely 
different feature(s) and these may dominate his solution. 5 Repair theory accounts for some bugs by 
hypothesizing that the student had/iot encountered the appropriate teaching necessary to perform 
the task. Suppose we make the converse assumption, that the appropriate teaching had been carried 
out, and further suppose that some students 6 do not gain competence in this domain by being loJd 
the rules but rather by inferring rules for themselves by noting the transformations which are applied 
to tasks by the teacher and in texts. 7 It seems reasonable that the student's inference procedure 
should be guided by his previous knowledge of the domain, in this case the number system, and that 
the student will normally infer several rules which are consistent with the example, and not just the 
"correct" rule. Indeed due to some missing knowledge the "correct" rule may not be inferred. (And 
so the fact that the student never uses the "correct" method along with several "buggy" methods is 
noi evidence that he has NOT encountered the material before). We shall refer to this process as 
Knowledge Directed Inference of Multiple rules, or mis-generalization for short. 

Suppose, the student saw the following stages in an algebraic simplification: 

3 * X * 6 = > X = 6/3 

Then he might infer 

X = RHS number/LHS number OR X = LARGER number/SMALLER number 

We will surmise how a student would use such a rule-set. We will suppose that the abler students 
actively experiment with different "methods", and use their own earlier examples, examples worked 
by the teacher and in the text to provide discriminatory feedback. From our experiment with 14-year- 
old students we have direct evidence that some students are aware of having a range of applicable 
rules and being unsure of when to select a particular method, Sleeman [1983a]. That study did not 
provided any insights into the rule-selection processes used by these students. We could suggest the 
common default, i.e., that the process is random. However, studies in cognitive modelling have 



Earlier Sleeman and Brown [1982} have argued: " Perhaps more immediately, il suggests that a Coach must pay 

attention to the sequence of worked examples, and encountered task states, from which the studenf is apl to abstract (invent) 
functional invanances This suggests that no matter how carefully an instructional designer plans a sequence of examples, he 
can never know all the intermediate steps and abstracted structures that a student will generate while solving an exercise. 
Indeed, tt.e student may well produce illegal steps in his solution and from these inveni illegal (algebraic) "principles". 
Implementing a system with this level of sophistication still presents a major challenge to ttv ITS/Cognitlve Science 
community... " 

g 

Note I am no/ claiming that there is a single mechanism. 

independently. VanLehn has come to a similar conclusion, the Sierra system described in his thesis relies heavily on 
inference, VanLehn [1963]. 
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already discredited this explanation many times, so we will postulate that the process is deterministic 
but currently "undetermined". It is further suggested that tasks which show a rule is inadequate will 
weaken belief in the rule, but once a (mal)-rule is created it may not be completely eliminated 
• particularly if the "counter-examples" are not presented to the student fcr some period. Thus given 
this view point, the phenomena of bug-migration occurs because the (less able) student has inferred 
a whole range of rules and selects a rule using a "black-box" process.. Given a further task, he again 
chooses a method and hence selects the same or an alternative algorithm, influenced partly by the 
relative strengths of the rules. That is if the relative weights are comparable, it is more likely that the 
student will select a different method for each task. If one weight "dominates" then it is likely that the 
corresponding method will be selected frequently. Further, if only one (mal) rule is generated by the 
induction process then this approach predicts that the student will consistently use that rule/ 

We suggest that many of the bugs encountered in the subtraction domain can be accounted for by 
this (inference) mechanism. For instance the Smaller-from-Larger bug, where the smaller number is 
subtracted from the larger independent of whether the larger number is on top or the bottom row, 
seems one such example, Brown & Burton [1978] and Young & O'Shea [1981]. Brown & VanLehn 
[1980] report that because borrowing was introduced, with one group of students, using only tasks 
with 2 columns, these students inferred that whenever borrowing was involved they should borrow 
from the left-most column, their "Always-Borrow-Left" bug. So it appears important to ensure that the 
example set includes some examples to counter previously experienced mal-rules. Indeed it seems 
as if task-sets can be damaging if they are too preprocessed and contain too little "intellectual 
ruffage"; Michener [1978] puts a similar argument. Additionally, Ginsburg [1977], quotes several 
instances of young children inferring the name "three-ty" for 30, given the names for "3", "4", "5", 
"40", "50", "60". So given the wealth of experimental evidence this alternative explanation should 
be given serious consideration. 

Further, I have two philosophical reservations about repair theory. Firstly, that by some mechanism 
not articulated all students acquire a common set of impasses, and moreover they consistently 
observe these. Secondly, repair theory which sets out to explain major individual differences at the 
task level, itself proposes a specific mechanism common to all students. 8 On the other hand, mis- 
generalization predicts that the individual's initial knowledge profoundly influences the knowledge 
which is subsequently inferred, and captures the sense in which learners are active theory builders 
trying to find patterns, making sense out of observations, forming hypotheses, and testing them out. 



4. Summary 

Firstly, there are two hypotheses which explain bug-migration the one given by repair theory and 
the one put forward here, namely mis-generalization. Of course it is possible that each may be 
applicable in different situations. Secondly, several "algorithms" have been presented for creating 
student models. I believe these are suggestive about the processes used when a student solves 
(these) tasks. Repair theory suggests that it can be explained by making "repairs" to incomplete 
core-procedures, whereas Young and O'Shea suggest that t is adequate to take a correct procedure 
and merely delete components. The data for the algebra manipulative mal-rules can be adequately 
explained by either. However, Young and O'Shea's approach r,eems inadequate to explain *he 



Indeed I am concerned that many theories of (child) development do not accept the possibility of there being significant 
individual differences in development, but merely in the individual's rate of progress and the level of his final maluration. 
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parsing mal-rules. Indeed, we have to extend revised repair theory More the results reported here 
can be accommodated. This paper claims that there are two very dif* tnt types of malrules at large 
with algebra students ■ namely manipulative and parsing mal-rules. ? his second category of 

algebra errors, and much of the data collected in other areas, s» •. .to be best explained by a 
further mechanism, namely mis-generalization. However, once inferred ■ believe rules are additionally 
applied incorrectly, and that the mechanism(s) described in Young & O'Shea, repair theory and 
section 2.1 , are appropriate for this stage. 
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